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Background: Telemedicine has been used for many years to support doctors in the developing world. Several 
networks provide services in different settings and in different ways. However, to draw conclusions about 
which telemedicine networks are successful requires a method of evaluating them. No general consensus or 
validated framework exists for this purpose. 

Objective: To define a basic method of performance measurement that can be used to improve and compare 
teleconsultation networks; to employ the proposed framework in an evaluation of three existing networks; to 
make recommendations about the future implementation and follow-up of such networks. 
Methods: Analysis based on the experience of three telemedicine networks (in operation for 7-10 years) that 
provide services to doctors in low-resource settings and which employ the same basic design. 
Findings: Although there are many possible indicators and metrics that might be relevant, five measures for 
each of the three user groups appear to be sufficient for the proposed framework. In addition, from the 
societal perspective, information about clinical- and cost-effectiveness is also required. The proposed 
performance measurement framework was applied to three mature telemedicine networks. Despite their 
differences in terms of activity, size and objectives, their performance in certain respects is very similar. For 
example, the time to first reply from an expert is about 24 hours for each network. Although all three 
networks had systems in place to collect data from the user perspective, none of them collected information 
about the coordinator's time required or about ease of system usage. They had only limited information about 
quality and cost. 

Conclusion: Measuring the performance of a telemedicine network is essential in understanding whether the 
network is working as intended and what effect it is having. Based on long-term field experience, the suggested 
framework is a practical tool that will permit organisations to assess the performance of their own networks 
and to improve them by comparison with others. All telemedicine systems should provide information about 
setup and running costs because cost-effectiveness is crucial for sustainability. 
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All doctors have their own personal network of 
contacts that they use when the diagnosis and 
management of cases exceeds the breadth of their 



own knowledge and experience. However, personal net- 
works are not always sufficient and have shown many 
limitations in practice (1). Thus, doctors often make use 
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of telemedicine networks that give access to a wider range 
of contacts than individual doctors commonly have 
available themselves. 

There is a long history of telemedicine networks being 
used to support doctors, particularly in low-resource 
settings, such as developing countries (2). Such telemedi- 
cine networks are used mainly for educational or clinical 
purposes. Networks that are used clinically generally 
provide diagnostic and management advice about specific 
patients and/or care problems (3). In the present article, 
we describe the characteristics of a form of telemedicine 
called teleconsultation (sometimes referred to as the 
provision of tele-expertise or second opinions). 

Among telemedicine networks with a clinical purpose, 
two fundamentally different models of operation exist. 
These correspond to the different ways in which an expert 
is chosen to reply to a query. In the first, a query is 
directed to a specific expert for reply; this might be called 
the targeted model. In the second, a query is directed to a 
group of experts, of whom all, some or none may reply; 
this might be called the open or distributed model. 
Examples of such networks are listed in Table 1 . 

Telemedicine appears to be useful in the context of the 
developing world (2) and store-and-forward teleconsulta- 
tion is one of the main applications used in this context 
(3). But which is the best model for teleconsultation? 
Which networks are performing well and which are not? 
To answer these questions, we need a framework for 
evaluating network performance. Frameworks exist in 
other domains (e.g. business, health improvement), but 
these are not really appropriate for our purposes and 
there does not appear to be a validated framework for 
evaluation and assessment of telemedicine systems. Thus, 
the objectives of the present study were: 

1. to define a basic method of performance measure- 
ment that can be used to improve and compare 
teleconsultation networks; 

Table 1. Telemedicine networks for low- and middle-income 
countries (second opinion teleconsultation networks 
providing services in the developing world with more than 
7 years of experience) 



Network 


Countries 


Model 


Swinfen Charitable Trust 


Various 


Targeted 


Ukraine 


Mainly Ukraine 


Targeted 


Institute of Tropical Medicine 


Mainly Africa 


Targeted 


iPath 


Various 


Open 


Partners Healthcare, Boston, 


Cambodia 


Targeted 


USA 






Tripler Army Medical Center, 


Mainly Pacific 


Targeted 


Honolulu, USA 


region 





2. to employ the proposed framework in an evaluation 
of three existing networks; and 

3. to make recommendations about the future imple- 
mentation and follow-up of such networks. 



Methods and findings 

In accordance with the three objectives, the work was 
conducted in three stages, as follows. 

Development of a framework 

To develop a suitable framework for measuring network 
performance, we first defined the concept. Then, we 
defined the actors. Finally, we defined the measurement 
perspective. This allowed the framework itself to be 
defined. 

Concept of performance 

Measuring the performance of a system is one aspect of 
its overall evaluation. A telemedicine network does not 
exist in isolation. It is a component or sub-system within 
an organisation. The performance of the organisation as 
a whole can be represented by its capacity to fulfil four 
functions: 

1. to realise its mission, for example to achieve its 
goals; 

2. to acquire resources, such as money and prestige, 
and adapt them to its needs; 

3. to produce services of appropriate quality and 
volume; and 

4. to maintain and develop its internal values, that is its 
organisational culture. 

In practice, there will be a dynamic tension between the 
achievement of these four functions (4). 

A telemedicine network can, therefore, be viewed as a 
tool for achieving the third of the four organisational 
functions. How then can the performance of such a 
system be measured? To evaluate any system requires: 

1. selecting the characteristics to be measured; 

2. choosing a suitable method to measure these 
characteristics; 

3. collecting the data; 

4. analysing the collected data; 

5. making decisions on the basis of the results; and 

6. implementing those decisions. 

Thus, performance measurement encompasses steps 
(2-4). It may involve measuring how the system works 
(i.e. process measures) or what it achieves (i.e. outcome 
measures). 

Process measures assess the general operation of the 
system. Here, the focus is on the ability of the system to 
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conduct its work. This may entail examining whether the 
necessary resources exist (e.g. funding, staff and equip- 
ment) to achieve the planned activities. A qualitative 
method is likely to be the most appropriate technique. 

Outcome measures assess the changes or benefits that 
result from the activities of the system. Typically, 
quantifiable measures are used to determine whether 
the system has achieved a certain level of performance. 

In the present context, the aim is to understand both 
whether the telemedicine network is working as intended 
and what effect it is having. Therefore, both process and 
outcome measures are required (even though outcome 
measures may be difficult to obtain in practice and need 
time before the effects can be measured). The choice of 
performance measures may be facilitated by considering 
a telemedicine network as a 'black box'. 

A black box is a device or system that can be viewed 
solely in terms of its input, output and transfer char- 
acteristics without any knowledge of its internal work- 
ings. Almost anything might be referred to as a black box, 
e.g. a transistor, an algorithm or a hospital. In the context 
of telemedicine network performance, the network (black 
box) has an input (the queries that are made) and an 
output (the answers provided), see Fig. 1. So that the 
black box can perform its function, resources have to 
be supplied. In the present context, these resources 
(principally the experts) are controlled by one or more 
telemedicine coordinators. 

The actors 

For the purpose of the present article, we describe the 
characteristics of a particular form of telemedicine called 
teleconsultation. The service is provided via a network 
operated by volunteers, although the principles are also 
relevant in networks operated by staff who are paid to do 
the work. Thus, we define a teleconsultation as taking 
place when: 



1. a requester, such as a referring doctor, requests 
information from an expert. In practice, the reques- 
ter makes the request to the telemedicine network, 
rather than to a specific expert; 

2. a telemedicine coordinator receives the request, 
evaluates it and decides which would be the appro- 
priate expert to answer it. (In the open model, the 
coordinator may allocate the request to a group of 
experts.) The coordinator then notifies the chosen 
expert that a query is waiting to be dealt with. Each 
request may generate multiple queries, for example if 
the first specialist to whom the query is sent is not 
available, then the query will need to be sent to 
another. In practice, because the coordinator is 
likely to require both medical and IT skills, the 
coordinator is often supported by one or more 
assistants; and 

3. an expert, such as a medical specialist, receives a 
request for information and (hopefully) responds to 
the requester. 

Thus, the present article is concerned with three types 
of persons (requesters, coordinators and experts) and 
how to measure their performance, although not in a 
punitive way. 

Measurement perspective 

In attempting to measure performance, it is helpful to 
consider whose perspective should be used. This is 
because the desirable performance from the point of 
view of the requester may be very different from that of 
the expert, or the coordinator. For example, the requester 
might hope for an immediate reply, whereas the specialist 
- who is likely to have many other demands on his or her 
time - may wish to deal with requests at leisure. 

User perspective. The first perspective is that of the 
network users, who all have the same desire: a reliable and 
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Fig. I. A telemedicine network viewed as a black box. 
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easy to use system, for which the minimum of training is 
required. Thus, the requester would like: 

1. a rapid answer; 

2. a definitive response to the questions posed ('defi- 
nitive' meaning comprehensive, appropriate and 
adapted to the local context); 

3. the possibility of discussing the case with the 
specialist, if the answer received is not definitive; and 

4. additional information, if relevant, such as refer- 
ences from the literature. 

The coordinator would like: 

1. a 'clean' referral, that is one without poor quality 
pictures or information contained in files of pro- 
prietary format. There may also be a requirement to 
receive carefully anonymised referral data if con- 
fidentiality is critical, as in humanitarian operations 
in war zones; 

2. access to a panel of experienced and available 
specialists, covering all sub-specialities; and 

3. a rapid response from the specialist, even if the 
response is only that a different specialist will be 
required. 

The expert would like: 

1. only appropriate referrals, that is requests that fall 
within his or her areas of expertise; 

2. sufficient information about the patient to permit an 
appropriate response to be provided; 

3. information about the local context to provide a 
reply that is well adapted to circumstances; and 

4. feedback from the referrer about the reply and/or 
subsequent follow-up data about the patient. 

Societal perspective. The other perspective that is 
relevant is that of society as a whole. Telemedicine 
is not simply a tool but a procedure allowing interactions 
between a network and the local health care systems. So, 
to measure the impact of the network, we must take into 
consideration the context, Fig. 2. Telemedicine may 
create a new way of accessing health care, for example 
obtaining a second opinion in remote areas or in a 
humanitarian setting, where the population often has no 
access to specialist physicians. This is a completely 
different matter from telemedicine that improves or 
optimises an existing health care system, for example 
telemedicine projects set up in industrialised countries. 
The distinction is important because it has direct 
consequences in terms of evaluation: in the latter case, 
a comparison between face-to-face and telemedicine 
consultation can be done, for example, whereas in the 
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former case, this is not possible because there is no 
previously existing reference system. 

The societal aims will, therefore, be a telemedicine 
network that provides clinically useful information at a 
'reasonable' cost. That is, the use of telemedicine will 
contribute to strengthening the local health care system. 

Patient's perspective. Finally, the patient's perspective 
should not be forgotten, as patient care is the ultimate 
goal of any telemedicine service. Even if the patient and 
requester both aspire to obtain the best possible health 
care management, their perception of what is best can 
differ. The doctor can take advantage of expert advice 
(e.g. from an educational point of view), whereas the 
patient would find no benefit at all. Patient satisfaction 
must, therefore, be assessed independently, and this 
requires a specific qualitative approach. Despite these 
potential differences between the views of requester and 
patient, we consider in the present context that the 
patient's perspective and the requester's perspective are 
tightly linked. 

The framework 

There is a wide range of indicators and metrics that might 
be relevant to the measurement of performance. Indica- 
tors 'indicate' impact but they do not attempt to quantify 
that impact, whereas metrics are 'numerical indicators' 
that allow the impact to be quantified. A combination of 
indicators and metrics is, therefore, needed. Because of 
the complexity of the environment in which telemedicine 
networks operate, both quantitative and qualitative 
methods are needed, and a multi-disciplinary approach 
is required for analysing results. In particular, a qualita- 
tive approach allows a deeper understanding of system 
use and, therefore, a better measurement of its true 
impact and so an improved assessment of quality. 

As far as the performance of the network as a whole is 
concerned, important metrics include: 

1 . the average speed of response to requests, that is, the 
average delay before a requester receives a definitive 
reply; and 

2. the cost of providing the service, for example, the 
coordinator's time, the infrastructure required. 

As far as performance of the network for the individual 
users is concerned, important metrics include: 

1 . the 'value' of the answer provided, for example, was 
it relevant (in accordance with the most up-to-date 
medicine adjusted by the expert's experience)? Did it 
help the patient? How long did it take? Did 
significant knowledge transfer occur between re- 
quester and expert (but also for the coordinators)? 
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ENVIRONMENT 




Fig. 2. Overview of a telemedicine system. Telemedicine networks are not isolated but located within larger health organisations 
that in turn are part of the general environment. 



Although a large number of possible indicators and 
metrics might be employed to answer these questions, in 
our experience five measures in each of the three user 
groups will suffice, that is, after discussion, it appears that 
these five measures capture the majority of the informa- 
tion required. 

Suggested performance measures 
There are basically five characteristics of the system that 
a requester needs to know about, in advance of submit- 
ting a new referral: 

1 . the rate at which queries are being submitted to the 
network (which tells the referrer how busy the 
network is and how well it is working); 

2. the proportion of failed queries (which tells the 
referrer what the chance of getting a reply is); 

3. the time to reply (which tells the referrer how long 
they will have to wait); 

4. the quality of the reply (which tells the referrer how 
good the replies are). The definition of 'quality' will 
include how useful the information is to the 
requester, how well adapted it is to the requester's 
local situation and also whether it follows the most 
recent guidelines as well as the educational value of 
the answer (knowledge transfer); and 

5. how easy the system is to use. 



The things that a coordinator needs to know are: 

1. How many questions will need to be handled? 
(Strictly speaking, what is the rate of arrival of 
requests for me to deal with?) 

2. How much time and effort will this entail? 

3. What resources are available to me? For example, 
how many experts are there, and what subject areas 
do they cover? What will I do if I do not have an 
appropriate expert for a particular query? Note that 
this leads to the question whether a more effective 
form of global telemedicine would be a network of 
networks. In other words, should there be a single, 
high-level clearing house for receiving telemedicine 
queries, from which queries could be directed to the 
participating networks as appropriate? 

4. Will I be able to obtain feedback from the users, for 
example, from the experts? Will I be able to obtain 
feedback on patient outcomes? 

5. How easy is the system to use? Are there likely to be 
IT problems and if so, how will these be solved? 

The things that an expert needs to know are: 

1 . How many queries will I receive? That is, what is the 
rate of work requests that I can expect? 

2. How much time and effort will be required to answer 
them? (Note that responding to a single query about 
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a clinical case may be much less demanding than 
answering a generic query that may require provid- 
ing the referrer with an informal summary of the 
literature.) How much time do I have to respond to 
the query? 

3. Will the queries be relevant to my area of expertise? 

4. Will I receive feedback about the patient outcomes? 

5. How easy is the system to use? For example, can I 
notify periods when I am not available to answer 
requests because of absence due to holidays or 
conferences? 

Some of the above items are aspects of quality, 
including the technical quality of attached images. 

Finally, from the societal perspective, the aspects of 
system performance that are important are: 

1. Is the network clinically effective? That is, does it 
improve patient outcomes? 

2. Is the network cost-effective? That is, is it an 
appropriate use of scarce resources? 

3. Are the network and its participants properly 
integrated into the relevant health care organisa- 
tions? 



Use of the framework 

To illustrate the use of the framework for measuring 
network performance, we applied it to the networks of 
our own experience. The authors are responsible for the 
operation of three telemedicine networks in different 
settings (low- and middle-income countries). The three 
networks all provide teleconsultation services, that is, 
they have a clinical purpose and have been in operation 
for periods of 7-10 years, see Table 2. 

Previous evaluations of the three networks 

Each of the three networks has been evaluated at various 

times during its operation: 

Swinfen charitable trust network. Type of evaluation 
conducted: Regular audits of system activity have been 
performed. Other forms of evaluation have included 
surveys of user satisfaction, analysis of submitted image 
quality and follow-up of patients. 

Main results. Audits of system activity show that the 
time required to provide a specialist's response has 
remained at 2 days or less over 10 years of operations 
(5-7); surveys of user satisfaction show that the referring 
doctors who responded made positive comments about 
the service and half said that it improved their manage- 
ment of the case (8); analysis of image quality showed 
that although average image size increased over a 4-year 
period, the quality of the submitted images did not (9); 
follow-up of patients showed that telemedicine had 
assisted with the diagnosis in all cases and the clinical 
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Table 2. Three second opinion networks that have operated 
for periods of 7-10 years 



1. Swinfen Charitable Trust network 

This service is provided by a UK charity and started operations in 
1999. It provides advice to doctors in developing countries using 
web-based store-and-forward messaging. In 2009, there were 
approximately 300 referrers from 45 developing countries. The 
advice is provided by a panel of about 400 consultants who 
donate their expertise to the charity. In the first 10 years of 
operation, the network handled approximately 2,000 telemedi- 
cine cases. 

2. Ukrainian Teleconsultation network 

This service started in 2000. This network is based in an expert 
trauma centre in Donetsk and provides treatment advice to seven 
community hospitals in the Donetsk region and four regional 
centres in the Ukraine using web-based store-and-forward and 
low-cost videoconferencing (Skype) methods. In the first 
1 0 years, more than 700 teleconsultations were conducted in 
trauma and orthopaedics. 

3. Institute of Tropical Medicine HIV/AIDS Telemedicine network 

This service is based at the Institute of Tropical Medicine in 
Antwerp and started in 2003. It provides second opinion advice 
about the use of antiretroviral therapy and AIDS care delivery 
using email and web-based store-and-forward methods. Expert 
advice from a network of 20 specialists in HIV/AIDS is offered to 
health care workers in resource-limited settings. In the first 7 
years, the telemedicine service provided over 1,000 teleconsul- 
tations to health workers in more than 40 countries, most located 
in sub-Saharan Africa. 



outcome for the patient was considered to be good in 1 5 
of the 44 cases (10). 

Ukrainian teleconsultation network. Type of evaluation 
conducted: A retrospective cohort study (n = 210) was 
carried out in 2007-2009. The aim was to define the 
quality of the teleconsultations and their influence on 
clinical outcomes. Patients were divided into three groups 
(telemedicine and two control groups) (11). 

Main results. The majority of teleconsultations (92%) 
had high relevance. The experts' recommendations had a 
real influence on the clinical strategy in 81% of the cases. 
There were more good clinical outcomes in the teleme- 
dicine patients studied than in a group of control patients 
(66% vs. 31%). An evaluation showed that telemedicine 
was associated with a sixfold increase in the odds of good 
clinical outcomes for trauma patients. 

Institute of tropical medicine HIV/AIDS telemedicine 
network. Type of evaluation conducted: A web-based 
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survey was sent to network users in 2006 and in 2008 to 
assess their perception of the system and its value in 
establishing the diagnosis, changing the management of 
care, providing reassurance to the referring doctor and 
any educational benefit (12). 

Main results. The referring clinicians reported that the 
service was beneficial in several ways: Establishing 
the diagnosis (42%), clinician's education (27%), litera- 
ture provision (14%) and for reassurance about decisions 
made prior to the consultation (12%). The service was 
judged to have been useful in influencing the management 
of the patients in 100% of cases. Most users (88%) 
reported that the service had significantly influenced the 
way they managed their cases and also other similar cases 
subsequently (13). 

Use of the performance framework 
The performance framework described above was applied 
to the three networks, using data available to the 
coordinators. The results are summarised in Table 3. 

Recommendations 

The following analysis is based on the experience of 
running three second opinion telemedicine networks for 
periods of 7-10 years and on an evaluation of the 
performance of the three networks using the proposed 
framework, that is, the aim is to offer evidence-based 
advice about practical implementation. 

Common features of the three networks 
Although the three networks are different in terms of 
their activity, their size and their objectives, their 
performance in certain respects is very similar. For 
example, the time to first reply from an expert is about 
24 hours for each network, see Table 3. However, in 
illustrating the use of the framework to assess the 
performance of the three telemedicine networks, it 
became clear that only some of the information required 
was in fact available. All networks had systems in place to 
collect certain data from the user perspective. None of 
them collected information about the coordinator's time 
required or about ease of system usage (for any of the 
three user types). They had only limited information 
about cost and quality. We recommend that networks 
collect such information routinely. 

System operation and usability 

Information about the coordinators' time was not 
collected by any of the networks. Nor was information 
collected about usability. Measuring ease of system usage 
requires data collection directly from the field to see 
how the system works in practice from the perspective 
of its users. This requires a qualitative approach, based 
on well-defined tools such as observation (participative 
or not), interviews (in depth, semi structured or guided 



discussion) or scale/questionnaires (14). Considering the 
major impact of the environment on system use, only 
such a pragmatic approach can provide insights into 
potential causal mechanisms of dysfunction and, there- 
fore, generate hypotheses to improve the system. Mixed 
methods have been used successfully in outcomes re- 
search in other disciplines (15) and, therefore, seem 
appropriate in telemedicine as well. We recommend that 
data on system operation and usability be collected 
routinely. 

Measurement of cost 

The issue of cost is less important in networks run by 
volunteers. However, estimation of costs is likely to be 
required in future operations where staff are reimbursed. 
Clearly, rigorous cost-effectiveness studies are best 
performed by an expert, for example, a health econo- 
mist^, 17), but some estimation of cost seems essential 
in the context of a telemedicine operation that relies on 
technology. A recent review of telemedicine in developing 
countries (3) showed that only a few articles provided 
such data although in the context of scarce resources, the 
appropriate allocation of resources is crucial (i.e. the 
consequences of wastage that will have little effect on 
health care in industrialised countries may have a 
profound impact in low-resource settings). Cost details 
also provide global information on the system, which can 
be useful for comparisons with other systems. We 
recommend that cost data be collected routinely. 

Assessment of quality of replies 

The quality of the replies was not systematically assessed 
by any of the three example networks - nor has it been 
reported more generally in other telemedicine networks - 
despite being fundamental to the main value of a 
telemedicine system. Content analysis of replies needs 
to be performed by independent experts, and ideally 
within the framework of a multi-disciplinary evaluation 
(medical, sociopsychological and anthropological) to 
take fully into account the environment and its impact 
on patient management. This kind of data collection has 
to be planned beforehand, which in turn emphasises the 
need for a standardised framework when setting up a new 
project. 

User satisfaction, including patient feedback is another 
aspect of the quality of the replies. Simple and reliable 
tools are available, although they may need to be adapted 
to suit the specific system being studied. The use of a 
numerical scale by Zolfo et al. (13). - that was based on 
the visual-analogue scale used for assessing pain - 
represents a useful example of the adaptation of validated 
tools for assessing user satisfaction. We recommend that 
quality of service data be collected routinely. 
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Table 3. Performance of three telemedicine networks 





SCT 




Ukraine 




ITM 




Performance 


Comment 


Value 


Comment 


Value 


Comment 


Value 


measure 
















Statistics for the year 




Statistics for the year 2007-2009 




Statistics from April 2003-March 






2009 








2009 




Requester's 














perspective 














1. Rate of query 




19.4/month 




5.8/month 




15.5/month 


arrival (new cases) 














2. Proportion of failed 




0 




0 




0 


queries 














3. Time to first reply from 


Median 


17 hours 




30 hours 




24 hours 


an expert 














4. Quality of replies 


Some evidence from a 


Not measured 


Some evidence from a research project (11). 


Not measured 


Some evidence from a research 


Not measured 




research project (10). 


routinely 


The quality of replies was measured 


routinely 


project (12) 


routinely 




Referring clinicians 




according to a specially developed scale 










reported that in 100% of 




(20). Ninety-two per cent of replies 










cases followed up 




{n = 1 32) were judged to be highly relevant 










(n = 22), the advice was 














useful 






















User satisfaction (38 respondents) 














measured on a scale 1-10. Utility of 














advice was rated 8-9 by 70% of the 














respondents 




5. Ease of system usage 




Not measured 




Not measured 




Not measured 


Coordinator's 














perspective 














1. Rate of query 


New queries, plus those 


37.7/month 


New queries 


5.8/month 


New queries. 


15.5/month 


arrival 


requiring 














additional responses 












2. Time required 




Not measured 




15-30 min 




Not measured 


3. Resources 




406 experts 




Five experts 




16 experts 


available 




available 




available 




available 


4. Feedback from 




Not measured 


Some evidence from a research project (11). 


Not measured 




Not measured 


experts/feedback on 






Feedback was available on the relevance of 


routinely 






patient outcomes 






the teleconsultation for 63% of the cases (n 









= 132) Feedback was available on patient 
outcomes for 33% of the patients {n = 1 32) 
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SCT 




Ukraine 




ITM 




Performance 


Comment 


Value 


Comment 


Value 


Comment 


Value 


measure 














5. Ease of system usage 




Not measured 




Not measured 




Not measured 


Expert's perspective 














1 . Rate of requests 




2.8/year per 




5.4/year per 




9.9/year per 


received (for those 




expert 




expert 




expert 


GA|Jfc!l Lb WML) [cOclvcU 














queries) 














2. Time required to 


Median time from 


1 1 .0 hours 




30 min 




Not measured 


answer 


allocation to response 












3. Relevance to own 




Not measured 




Not measured 




Not measured 


expertise 














4. Feedback on 


Some evidence from a 


Not measured 


Some evidence from a research 


Not measured 




Not measured 


patient outcomes 


research project (1 0) 


routinely 


project (11) 


routinely 






5. Ease of system usage 




Not measured 




Not measured 




Not measured 


Societal perspective 














1. Clinical- 


Some evidence from a 


Not measured 


Some evidence from a research project (11). 


Not measured 


Some evidence from a research 


Not measured 


effectiveness 


research project (10). 


routinely 


Referring clinicians reported that 


routinely 


project (13). Referring clinicians 


routinely 




Referring clinicians 




telemedicine produced satisfactory 




reported that telemedicine was 






reported that the 




outcomes in 22% of cases (n = 132) 




useful in influencing the 






telemedicine advice was 




and good or excellent outcomes in a 




management of their patients in 






useful in 21 of 22 cases 




further 76% 




100% of cases [n = 37) 






followed up, and the 














outcome for the patient 














was good in 1 5 cases 












2. Cost-effectiveness 




Not measured 




Not measured 




Not measured 


3. Integration into the 




Not measured 




Not measured 




Not measured 



health care system, for 
example, involvement of 
local people 



Richard Wootton et al. 



Societal perspective 

As well as an absence of data about the user perspective, 
the three networks collected very little information about 
network performance from the societal perspective. In a 
study of telehealth adoption in Quebec hospitals, Gagnon 
et al. (18) found that the local context was a crucial factor 
in long-term success. Understanding the local context 
requires a field approach and feedback from local users. 
Close collaboration with local national health care 
workers is necessary to make sure that any proposed 
telemedicine system will address real needs and meet local 
priorities. 

Only limited data were available on clinical effective- 
ness and none at all on cost-effectiveness. Measuring 
health outcomes (e.g. mortality and morbidity, diagnostic 
accuracy) is difficult in resource-limited settings, espe- 
cially in situations where health is influenced by a range 
of factors that are difficult to control for. Measuring cost- 
effectiveness is, therefore, likely to be extremely difficult, 
although some limited information about costs might 
be obtainable. We recommend that data be collected 
routinely to enable measurement of clinical- and cost- 
effectiveness. 

Implementation recommendations 
Based on our experience, the targeted model of tele- 
medicine network operation appears to be successful and 
sustainable. The three networks we have used to illustrate 
the proposed framework are different in terms of their 
activity, their size and their objectives, but their perfor- 
mance in certain respects is very similar. On the other 
hand, we cannot conclude that the targeted model is 
superior to the open model without further evidence such 
as performance measurements from a wider range of 
networks. 

Discussion 

A telemedicine network can be regarded as an example of 
a 'diffuse technology', rather like a digital picture- 
archiving system (PACS). As was pointed out during 
the evaluation of early PACS networks, there are ways of 
combining individual studies to arrive at strong evidence 
about the impact of a diffuse technology (19). For 
example, information from individual case studies can 
be combined, so that even if they represent relatively 
weak evidence individually, the aggregated value is 
strong. Thus, two or three separate case studies based 
on different data sources that show similar effects will 
help to confirm the value of a diffuse technology. Our 
hypothesis is that stronger evidence about the value of 
telemedicine networks in the developing world can be 
obtained from studies of multiple networks than from a 
study of any network alone. 

We have proposed a performance measurement frame- 
work for assessing telemedicine networks that provide 
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teleconsultation services to doctors in resource- 
constrained settings, although the principles are more 
general and could be applied to other telemedicine 
networks as well. The suggested performance measures 
for the three types of users show certain common, cross- 
cutting themes. All users are concerned with: 

1. Access (how busy is the network? what will my 
workload be?). 

2. Quality (how good are the responses? can feedback 
be obtained about patient outcomes?). 

3. Usability (how easy is the system to use? what will I 
do if there are technical problems?). 

Similar themes can be seen in the societal perspective. 

The framework is based on a core set of indicators and 
metrics. We have illustrated its use to assess the perfor- 
mance of three mature telemedicine networks. The results 
help us to understand whether these telemedicine net- 
works are working as intended and the effect that they are 
having. 

Telemedicine is a well-known minefield. It is a recipe 
for disaster to begin a pilot project without having a 
satisfactory method for evaluating network performance. 
We recommend that everyone uses a common framework. 
This will facilitate comparisons between networks. More 
important, it allows those responsible to make long- 
itudinal comparisons within their own networks, for 
example, to assess the effects of improvements. We also 
recommend that all networks collect basic information 
about their costs. 



Conclusion 

Measurement of performance is a fundamental aspect of 
network evaluation, yet there is no existing framework in 
the literature that is relevant in the present context. 
Conceptualising a telemedicine network as a 'black box' 
allows the identification of important indicators and 
metrics that are common to the different perspectives. 
While examining these different perspectives (user and 
societal), we propose a framework for performance 
evaluation based on five main items from each perspec- 
tive. This framework was applied to three different 
telemedicine networks, and certain common performance 
characteristics emerged. It also became clear that some 
important data were not measured at all, possibly because 
acquiring this information would require a specific field 
approach. 

For a comprehensive evaluation that takes into 
account the environment, mixed methods are required, 
which combine both quantitative and qualitative meth- 
ods. Cost-effectiveness analysis must be performed by an 
expert, but all telemedicine systems should at least 
provide information about setup and running costs. 

Citation: Global Health Action 2011, 4: 7214 - DOI: 10.3402/gha.v4i0.7214 



Experience with low-cost telemedicine in three different settings 



The present article describes the performance of three 
different telemedicine networks, operating in low- and 
middle-income countries. The performance framework is 
based on our experience. No telemedicine network should 
be started without having evaluation built in. Our 
suggested framework will permit an organisation to 
understand both whether the telemedicine network is 
working as intended and what effect it is having. 
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